Debiasing Evaluations That Are Biased by Evaluations

It is common to evaluate a set of items by soliciting people rate them. For example, universities ask students the teaching quality their instructors, and conference organizers authors submissions reviews. However, in these applications, often give higher rating course if they receive grades course, reviews papers are accepted conference. In this work, we call external factors "outcome" experienced people, consider problem mitigating outcome-induced biases given ratings when some information about outcome available. We formulate as known partial ordering on bias. propose debiasing method solving regularized optimization under constraint, also provide carefully designed cross-validation that adaptively chooses appropriate amount regularization. theoretical guarantees performance our algorithm, well experimental evaluations.